Beyond LLM Judges: Deep Evaluation for Conversational Search

As conversational search becomes mainstream, evaluating its performance requires more than simple LLM-as-a-judge approaches. This session presents our evaluation framework for the conversational search assistant at MediaMarkt, currently in A/B testing.

Our system uses an AI agent that decides which internal tool to call, interprets user queries, extracts search terms and filters, and calls our search API (lexical + semantic). While traditional evaluation might only assess the final response, we developed a multi-layered framework that evaluates search term extraction accuracy, facet match rate, no-result scenarios, search result relevance, multi-turn conversation quality, context consistency and overall relevance.

This talk shares real-world challenges such as handling ambiguous queries, measuring filter accuracy, calibrating automated judgments with human review and selecting LLMs for evaluation. The focus is on what proved useful, what didn’t, and how this shaped our conversational search system.

Ruchi Juneja

MediaMarktSaturn

Ruchi works as a Sr. Data Scientist at MediaMarktSaturn in the Search ML team. For over a decade, she has immersed herself in the world of data science working on various business problems. The last eight years have truly defined her path to the fascinating realm of search. Her passion for search technologies has led her on a journey across various industries, from education to music and e-commerce, delving deep into the intricacies of search algorithms, specialising in vector search, search ranking and query understanding.